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analyze  the  simulated  data.  The  results  revealed  the. t  (1)  wail e 
synergisms  produce  notable  effects  throughout  the  range  of  intercorreiation 
values,  the  most  striking  influence  was  found  for  negative  intercorrelations, 

(2)  descriptive  indices  (such  as  R“)  were  relatively  unrevealing  about  the 
presence  of  synergisms,  (3)  in  contrast,  inferential  indices  (particularly 
the  hierarchical  test)  were  much  more  revealing,  and  (4)  there  were  cor.sicera'oie 
between-stimulus  differences  in  terms  of  how  easily  synergisms  could  be 
accurately  detected.  These  findings  imply  that,  because  synergisms  have  .  een 
frequently  overlooked  by  investigators,  they  should  be  specifically  testeu 
for  in  future  research  using  nonorthogonal  designs. 
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Detection  of  Multiplicative  Synergisms  in 
Simulated  Data  for  N'onorthogonal  Designs: 

What  Lies  Beyond  Linearity? 

According  to  Webster's  (1976)  dictionary,  a  synergism  is  an  action 
"such  that  the  total  effect  is  greater  than  the  sum  of  the  effects  taken 
independently."  In  psychological  terms,  synergisms  are  usually  represents! 
by  a  multiplicative  interaction  between  two  or  more  variables.  For  instance, 

R  =  f  (A  x  B),  (1) 

where  the  response  R  is  a  function  of  the  multiplicative  combination  of 
variables  A  and  B.  A  number  of  behavioral  models  and  theories  take  the  form 
of  Equation  1.  Examples  of  synergisms  in  psychology  can  be  found  in  learning 
(performance  =  drive  x  habit  strength),  industrial  (performance  =  ability  x 
motivation),  and  social  (level  of  aspiration  =  desirability  of  goal  x  expectancy 
of  success).  For  a  more  complete  discussion  of  the  importance  of  synergism, 
along  with  numerous  psychological  examples,  see  the  review  by  Shanteau  (1981). 

In  experimental  designs  which  are  orthogonal,  such  as  factorial  designs, 
detection  of  a  mulitplicative  synergism  is  relatively  straightforward.  In 
analysis  of  variance  terms,  a  synergism  will  lead  to  an  interaction.  More¬ 
over,  the  interaction  will  take  a  specific  bilinear  form,  i.e.,  the  data 
will  plot  as  a  diverging  fan  of  lines.  While  several  statistical  procedures 
are  available,  the  most  generally  useful  technique  is  to  evalute  the  linear  x 
linear  component  of  the  interaction.  The  various  procedures  are  discussea 
and  compared  in  Shanteau  (1978,  1981). 

In  contrast  to  the  orthogonal  case,  the  detection  of  synergisms  in 
nonorthogonal  designs  is  much  less  clear  cut.  Nonorthogonal  designs  typically 
arise  when  the  levels  of  variables  are  assigned  in  some  non-control led  fashion, 
e.g.,  as  in  a  representative  design  3runswih,  1956).  Typically,  this 
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means  that  the  variables  will  be  int ercor related  with  one;,  ot,.,r.  '.he 
problems  caused  by  such  intorcorrcl at ions  arc  well  recognized  in  ti.e 
literature  on  multipl  e-regre.ssion  analyses  (e.g.,  Darlington,  1969). 

However,  the  difficulties  introduced  by  intercorrelation  in  the  detection  of 
synergisms  have  yet  to  be  analyzed.  Therefore,  the  purpose  of  the  present 
study  is  to  evaluate  the  effects  of  synergisms  on  nonorthogonal  designs. 

Research  Strategy 

While  it  would  be  desirable  to  use  real  data  sets  for  analyzing  synergisms, 
this  is  impractical  on  two  grounds:  (a)  It  is  difficult,  if  not  impossible, 
to  know  in  advance  whether  a  synergism  does  or  does  not  exist  in  a  given 
set  of  data.  Without  such  knowledge,  any  further  analyses  would  be  fruitless 
for  present  purposes.  (b)  There  is  no  feasible  way  to  obtain  data  sets  which 
correspond  to  all  the  conditions  which  would  be  desirable  to  examine.  More¬ 
over,  any  real  data  sets  are  likely  to  have  been  influenced  by  a  variety 
of  other  unknown  and  probably  idiosyncratic  factors. 

Because  of  these  problems,  an  alternative  research  strategy  was  developed 
based  on  the  construction  of  simulated  data  sets  There  are  three  note¬ 
worthy  features  to  such  data  sets:  First,  the  presence  (or  absence)  of  a 
synergism  can  be  built  into  simulated  data.  That  is,  a  synergism  can  be 
specifically  included  or  excluded  in  the  simulated  data-generating  mouel. 

Thus,  the  "truth"  about  the  simulated  data  is  known  a  priori. 

Second,  various  properties  of  an  experimental  design  can  be  easily 
and  systematically  manipulated  with  simulated  data.  For  instance,  the 
degree  of  cue  intercorrelation  between  two  variables  can  be  varied  from 
high  positive  to  high  negative  with  numerous  steps  in  between.  Other  proper¬ 
ties,  such  as  the  size  of  the  design  can  also  be  easily  specified.  There¬ 
fore,  an  advantage  of  simulated  data  is  that  potentially  important  properties 
of  the  design  can  be  varied  in  predetermined  ways. 
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Third,  nonor thogonal  designs  with  the  same  properties,  car  be  produces 
in  many  different  ways.  That  is,  a  variety  of  stimulus  sets  can  have  the 
same  degree  of  intercorrelation  values,  etc.  This  lack  of  uniqueness  in 
nonorthogonal  designs  makes  it  desirable  to  compare  alternative  stimulus 
sets  with  the  same  correlations.  Thus,  simulated  data  sets  allow  a  direct 
way  to  evaluate  the  consistency  of  any  results  involving  synergisms . 

Of  course,  the  use  of  simulations  also  has  its  disaavantages .  Some  o* 
these  will  be  taken  up  in  more  detail  in  the  Discussion.  .  -  this  point, 
however,  it  is  worth  emphasizing  that  every  effort  was  made  to  produce  du t~ 
sets  which  "look  like"  real  data.  Towards  this  end,  a  variety  of  models 
were  used  which  resemble  those  which  are  known  to  be  used  by  subjects.  In 
addition,  error  was  added  to  the  data  at  levels  which  are  similar  to  the 
values  found  in  typical  response  sets.  In  short,  the  simulated  data  had 
all  the  outward  appearances  of  realistic  data. 

The  remaining  sections  of  the  paper  begin  with  a  detailed  description 
of  the  simulation  technique  used.  This  also  includes  a  consideration  of 
the  procedures,  analyses,  etc.,  employed.  Then  the  results  from  the  analyses 
of  over  14,000  simulated  data  sets  are  presented.  Finally,  the  last  section 
contains  a  discussion  of  the  implications,  as  well  as  qualifications,  of  the 
present  results. 

Simulation  Approach 

The  basic  goal  behind  the  present  research  approach  was  to  separate  a 
typical  experimental  study  into  three  stages  and  to  simulate  each  stage 
separately.  As  shown  in  Table  1,  these  stages  correspond  to  the  construction 
of  an  experimental  design  (the  environment),  the  formation  of  the  responses 
according  to  some  strategy  (the  subject),  and  the  analysis  and  interpretation 
of  the  results  (the  experimenter).  Since  independent  algorithms  were  constructed 
for  each  stage,  the  simulation  techniques  will  be  considered  separately. 

i 


Insert  Table  I  about  here 
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Environmental  Simulation 

The  experimental  design  specifies  the  stimulus  environment  m  which  re¬ 
search  evidence  is  collected.  Obviously,  a  subject  can  only  provide  rea/.o.'.Se 
to  the  particular  stimulus  combinations  presented.  This  means  that  the 
experimental  design  can  play  a  crucial  role  in  determining  whether  a 
specific  relation,  such  as  a  synergism,  can  or  cannot  be  detected  in  the  aita 

In  the  present  case,  the  construction  of  each  stimulus  design  involved 
a  two-step  process.  The  first  step  was  based  on  tentatively  constructing  a 
stimulus  set  intended  to  reflect  various  prespecified  conditions.  The 
second  step  was  based  on  examining  the  tentative  set  to  see  if,  in  fact, 
the  desired  conditions  had  been  met.  For  instance,  if  a  prespecified  level 
of  cue  intercorrelation  was  desired,  then  the  observed  intercorrelation  value 
for  a  tentative  set  was  compared  to  the  desired  value.  If  the  stimulus  set 
met  all  the  conditions ,  then  it  was  used.  If  not,  and  if  minor  adjustments 
did  not  produce  a  satisfactory  stimulus  set,  then  the  set  was  discarded  and 
the  process  started  over.  Both  of  these  steps  will  now  be  considered  in  more 
detail. 

Cue  generation.  A  computer  algorithm  incorporated  in  the  program 
CUEGEN  (see  the  Appendix  for  details)  was  used  to  construct  sets  of  nonorzhog 
onal  stimuli  with  specifiable  properties.  Some  of  these  properties  were 
arbitrarily  fixed  for  purposes  of  this  research  project,  while  others  were 
varied.  The  following  four  properties  were  held  constant  across  all  stimulus 
sets:  (1)  Only  two-cue  stimulus  sets  were  used  (see  Table  2  for  an  example). 

This  allowed  complete  freedom  to  vary  intercorrelations  from  +1  to  -1;  the 
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use  of  three  or  inure  cues  would  have  reduced  the  range  of  correlation  values 
possible.  (2)  The  range  of  the  cue  values  was  restricted  from  0  to  100. 

Tliis  restriction  was  purely  for  convenience  and  involves  no  loss  of  generality 
(3)  A  uniform  sampling  distribution  was  specified  for  each  cue.  Although 
other  distributions  such  as  normal  were  considered,  it  proved  to  be  easier 
to  construct  appropriate  stimulus  sets  using  the  uniform.  (4)  The  .r.ean  jr.- 
standard  deviation  were  specified  to  be  50.0  and  20.0,  respectively .  As 
the  case  for  the  other  fixed  properties,  these  arbitrary  va  ies  aopearc- 
to  have  little  impact  on  the  pattern  of  results  observed. 


Insert  Table  2  about  here 


There  were  three  properties  which  were  varied  systematically  in  the 
construction  of  the  stimulus  sets:  (1)  The  intercorrelation  values  were 
specified  to  be  +.90,  +.75,  +.50,  +.25,  .00,  -.25,  -.50,  -.75,  and  -.90. 

These  values  both  covered  the  range  of  possible  intercorrelations  and  were 
reasonably  dense  and  well-spaced.  (2)  The  number  of  stimuli  in  each  stimulus 
set  was  specified  to  be  either  25  or  100.  These  numbers  are  representative 
of  what  is  typically  used  in  "small"  and  "large"  nonorthogonal  designs. 

(3)  Nine  independent  stimulus  sets  were  generated  for  each  combination  of 
correlation  value  and  stimulus  set  size.  There  were  thus  a  total  of  9 
(intercorrelations)  x  2  (stimulus  set  sizes)  x  9  (independent  sets)  =  162 
stimulus  sets  generated. 

Testing  the  stimulus  sets.  Each  constructed  set  was  subjected  to  a 


series  of  tests  to  check  whether  it  was  close  to  the  desired  properties, 
such  as  the  specified  intercorrelation  value.  Only  if  a  stimulus  set 
satisfied  all  the  tests  was  it  kept.  Otherwise,  the  cue  values  were  randomly 
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adjusted  and  the  "out  repeated.  If  -if ter  five  adjustments,  the  at* t 

was  still  not  satisfactory,  then  it  was  discarded  and  a  new  set  was  gener.. tee 
using  the  CUEG’KN  program. 

There  were  four  types  of  tests  that  each  stimulus  set  had  to  pass  at 
the  .05  level  in  order  to  be  acceptable:  (1)  The  intercorrelation  between 
the  cue  values  (i.e.  ,  between  paired  entrees  in  Table  2)  had  to  be  non  si  ,..n x- 
cantly  different  from  the  desired  value.  (2)  The  mean  for  each  cue  u-  .,  .•.-.ciy 
(i.e.,  each  column  in  Table  2)  had  to  show  nonsignificant  i__veat_C'U_  :.u.. 
the  desired  value.  (3)  Similarly,  the  standard  deviation  for  each  cue  ...... 

to  be  nonsignif icantly  different  from  the  specified  valve.  (4)  finally, 
the  distribution  of  values  for  each  cue  separately  had  to  approach  the 
uniform  distribution.  (To  make  this  test,  each  distribution  was  dividoc. 
into  segments  and  deviations  from  the  uniform  were  computed  for  each  segment. 

A  chi-square  procedure  was  then  used  to  test  for  any  discrepancies.) 

In  ail,  each  constructed  stimulus  set  had  to  pass  seven  tests:  one  for 
the  intercorrelation  value  and  two  each  for  means,  standard  deviations,  and 
uniform  distribution.  In  practice,  almost  all  stimulus  sets  satisfied  the 
intercorrelation  restriction.  Tests  on  means  and  standard  deviations  resulted 
in  a  few  rejected  sets.  However,  the  distribution  test  was  the  most 

demanding  and  produced  by  far  the  highest  proportion  of  rejected  sets. 

However,  it  was  possible  in  all  cases  to  construct  nine  independent  stimulus 
sets  which  satisfied  each  of  the  tests. 

Subj ec t  Simulation 

To  simulate  the  subjects'  behavior  in  an  experiment,  a  two-step  process 


was  followed.  First,  a  model  was  specified,  e.g. ,  multiplying,  for  each 
simulated  subject.  This  model  represents  the  "truth"  to  be  detected  by  the 
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subsequent  analysis.  Second,  random  error  was  introduced  io  . 

realistic  responses.  Each  of  these  two  steps  will  now  oe  con.,  .acre...  ir. 
detail . 

Model.  Three  different  models  were  used  in  these  simulations.  Tht 


first  was  a  multiplying  model  in  which  the  two  cues,  X.  unu  X.  ,  w<.  •/<.  com:.. 

- - ‘ - —  X 

as  follows: 


Y  =  X  x  X.. 
m  1  2 


For  the  example  shown  in  Table  2,  the  first  pair  of  valuer  ...  X. 
67.4  and  44.3,  respectively.  The  product,  Y  ,  wouic  be  67.4  x  44.3 
This  model  represents  a  "pure  synergism"  in  the  form  of  cross-prod-: 
The  second  model  involved  adding  the  two  cues: 


Ya  "  X1  +  V 


For  the  first  pair  of  values  inTable2,  Y  would  be  67.4  4-  44.3  -  111.7. 

a 


model  provides  a  baseline  or  control  condition  in  which  a  synergism  is  known 
not  to  exist. 

The  third  model  was  a  combination  adding-multiplying  model: 

(4) 


Yc  =  Xi  +  x2  +  (Xx  x  X2). 


For  the  example  above,  Y  would  be  67.4  +  44.3  +  (67.4  x  44.3)  =  3097.5. 


This  combination  model  allows  examination  of  a  synergism  in  the  context  o;  ■. 
adding  process. 

Error.  To  produce  realistic  data,  error  obviously  must  be  introduce- . 
In  considering  error,  two  major  choices  have  to  be  made  which  correspond  to 
the  location  and  the  size  of  the  error  term.  In  regard  to  location ,  c.v.  . 
can  be  added  either  before  or  after  the  cue  values  are  combined  in  Equation:* 
2  to  4.  Introducing  error  before  the  combination  process  corresponds  tc 
variability  produced  by  stimulus  or  perceptual  uncertainty.  Introducing 
error  after  the  combination  corresponds  to  v-.. lability  caused  by  response  or 


output  uncertainty. 
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v or  two  reason:;,  it  was  decideu  .c  00;.  c»Ti.”  ;  . 

stimulus  values.  Thar  is,  random  error,  11,  was  ad.!,  ,:  r  ..  i-a,  V  value  ... 

Equati ons  2  to  4  so  tent  the  stmi i ,itcu  dm.,,  Y'  ,  c,,n  0e  ,  as: 

Y’  =  Y  +  E.  (5; 

The  first  reason  is  that  statistical  models  typically  as.  a  arse  a  pcat-.:or..biru'.t  --,.1 

additive  error.  Since  one  of  the  goals  of  this  study  was  to  ovtlt.uC  _ 

statistical  techniques,  it  seemed  preferable  tc  make  the  data  consistent  wi ch 
assumptions  made  by  the  statistical  analyses. 

A  second  reason  is  that  there  is  some  limited  empirical  support  for  .  . 
error-after  location.  Shanteau  (1970)  examined  within-cell  vanabil  ...y 
in  a  task  known  to  produce  synergistic  behavior,  i.e.  ,  a  gambling  task. 

The  cell  variances  were  found  to  be  unrelated  to  the  cell  means;  this  is 
consistent  with  Equation  5,  but  inconsistent  with  a  bef ore-combination  of 
error . 

The  other  major  choice  involves  the  size  of  the  error  term  to  be  acced. 

It  should  be  clear,  since  the  ranges  of  the  Y  values  for  the  three 

models  are  quite  different,  that  the  same  size  error  ernnot  be  used  for  ail 
three  models.  Instead,  the  size  of  the  error  was  calibrated  individually 
to  reflect  each  set  of  Y  values.^  Thus,  the  Y’  values  were  produced  by  adding 
to  each  Y  a  random  normal  error  value,  with  mean  zero  and  standard  deviation 
equal  to  c.  After  trial-and-error  exploration,  it  was  found  that  a  c  value 
equal  to  one-half  of  the  coefficient-of-variation  for  the  values  produced 
the  most  reasonable  looking  results,  i.e,,  c  =  U  x  Standard  Deviation  t  Mean. 

A  variety  of  other  ways  of  defining  c  were  explored,  and  in  general  the  pact, rn 
of  the  results  did  not  appear  to  depend  on  the  definition  of  c  used.  (Sem 
additional  comments  on  issues  related  to  error  appear  in  the  Discussion). 

Using  the  approach  outlined  above,  ten  i -dependent  data  sets  were  generated 
for  each  stimulus  set.  In  other  words,  from  each  set  of  Y  values,  different 
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combinations  of  random  error  were  added  to  forir.  ton  set  of  Y'  In 

summary,  ten  simulated  subject;;  were  created  for  each  of  tbe  V  sets  of 
stimulus  values. 

Expel* i mentor  Slmul a t ion 

For  present  purposes,  the  experimenter 's  role  is  that  of  analyzing  arm 
comparing  the  results  for  various  statist. cal  methods.  Since  there  are  .. 
variety  of  methods  which  can  be  used,  the  experimenter's  question  becomes: 
which  of  the  various  statistical  techniques  is  most  sensxt.  v  to  the  preset, 
(or  absence)  of  a  synergism?  Before  dealing  with  this  question,  however,  t 
will  first  be  necessary  to  review  the  overall  multiple-regression  approach 
used . 

Multiple-regression  analysis.  Each  of  the  simulated  subjects  (i.e. , 
each  set  of  Y'  values)  was  analyzed  by  three  types  of  multiple-regression 
models.  These  models  correspond  to  the  three  models  used  to  create  the 
data  in  Equations  2,  3,  and  4.  Thus,  each  data  set  was  analyzed  using  a 
multiplicative  or  pure  cross — product  regression  model: 

\  =  (Xx  x  X2),  (6) 

where  Ym  is  a  predicted  value  derived  from  the  product  of  the  cue  values 
X^  and  X2>  Similarly  each  data  set  was  analyzed  using  an  additive  or  linea 
regression  model: 

Y  =  6,  X,  +  S0  X„,  (7) 

a  11  l  l 

where  the  predicted  value  Yfl  is  obtained  from  a  weighted  sum  of  the  stim...,. 
values.  Finally  each  data  set  was  analyzed  by  a  multilinear  (combined 
additive-multiplicative)  regression  model: 

Yc  =  Xx  +  32  X2  +  63  (X1  X  X2),  (b) 

where  the  predicted  Y^  value  is  a  weighted  sum  of  the  various  terms. 
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In  the  actual  analyses,  ordinary  least  squares  procedures  _ a.  .:soo  to 

obtain  estimates  of  the  S  weights.  These  estimates  minimize  the  discrepancy 
between  the  original  Y'  values  and  the  derived  Y  values.  The  raw  regress .or. 
analyses  were  performed  on  the  data  for  each  simulated  subject.  For 
convenience,  however,  the  results  have  been  averaged  across  the  ten  subjects 
generated  for  each  stimulus  set.  (As  an  aside,  the  zero  intercept  and 
weight,  PQ  ,  have  been  omitted  for  clarity  from  Equations  6  to  8.  These  vult. 
were  invariably  near  zero  and  contributed  nothing  to  the  in  orpretacicn  of  c 
results. ) 

Statistical  indices.  Based  on  the  multiple-regression  analyses,  a  r»u::.b 
of  statistical  measures  were  computed.  These  measures  are  considerec  in 
some  detail  in  the  Appendix.  However,  because  the  results  for  many  of  the 
indices  were  redundant,  only  the  most  relevant  measures  will  be  considerec 
in  the  results.  These  indices,  which  are  listed  across  the  top  of  Tables 
3  and  4,  will  be  described  as  necessary  in  the  Results  section. 

Results 

Since  14,580  separate  multiple-regression  analyses  were  run,  it  is 
obviously  necessary  to  be  highly  selective  in  presenting  results.  For 
brevity,  graphical  summaries  will  be  presented  of  just  the  most  relevant 
statistical  results.  In  addition,  only  short  descriptions  will  be  given  of 
the  indices  (see  the  Appendix  for  a  more  complete  description 

of  the  statistical  procedures). 

Descriptive  Indices 

The  squared  multiple-correlation  value,  R^,  provides  a  standard  measure 
of  the  variance-accounted-for  by  a  regression  model.  Figure  1  shows  the 
average  R^  values  for  the  fit  of  an  additive  (linear)  regression  model 
(Equation  7);  the  left  panel  gives  the  results  for  the  25-stimulus  condition 
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and  the  right  panel  gives  the  result:,  for  the  IGO-sti.m.iu.,  corbie i  *n . 
three  lines  in  each  panel  correspond  to  the  three  da  t a-geno rati up,  .notei.. 
Equations  2  to  4  ,  wi  th  the  nine  levels  of  cue  intercorrelation  listed  a  Ion. 
the  horizontal  axis.  Each  of  the  plotted  values  in  Figure  1  is  the  average 
of  90  multiple-regression  analyses:  ten  simulated  subjects  in  each  of  nine 
stimulus  sets. 


Insert  Figure  1  about  here 


Three  general  observations  follow  directly  from  this  and  the  other 
figures.  First,  the  results  appear  to  be  reasonably  smooth  and  lawful. 
Moreover,  several  trends  both  between  curves  and  across  correlation  value* 
are  easily  discernable.  Thus,  at  least  at  a  surface  level,  the  present 
simulation  approach  has  produced  orderly  data. 

Second,  the  multiplying  data  model  (F.quation  2)  and  the  adding-multiplying 
model  (Equation  4)  lead  to  essentially  identical  results.  With  few  exceptions, 
the  statistical  indices  produced  by  these  two  models  are  virtually  indistin¬ 
guishable.  This  suggests  that  what  a  synergism  is  combined  with  is  not  as 
important  as  the  fact  that  a  synergism  is  present. 

Third,  the  adding  uata  model  (Equation  3)  led  to  results  which  are 
consistently  lower  than  the  results  for  the  other  two  data-generating  moe*..-,. 

In  some  circumstances,  this  might  be  expected  since  the  adding  model  serve* 
as  a  baseline  for  many  of  the  analyses.  For  indices  such  as  R“ ,  however , 
the  iower  values  were  unexpect  i.  The  reason  for  this  result 

apparently  lies  in  the  procedure  used  to  add  error  to  the  simulated  data. 

Using  the  coefficient  of  variation  to  determine  the  size  of  the  error  see... 
to  have  introduced  relatively  more  error  into  the  adding  data  than  into  the 
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other  two  data  models.  Hence,  the  k*  are  proportionally  1  owe r .  Horton;. tel y  , 

this  difference  does  not  influence  any  of  the  major  findings  reported  here 

(although  it  docs  suggest  that  some  attention  be  given  to  the  issue  of 

2 

compatible  error  values  in  any  future  simulation  research). 

One  unique  finding  in  Figure  1  is  that  the  results  in  the  two  panels 
are  remarkably  similar.  For  instance,  the  curves  for  the  synergistic  r/.o  u  >- 1  _ 
are  nearly  straight  for  positive  intercorrelations  with  a  gradual  increase  .o 
around  .70.  Thus,  for  positively  correlated  cues,  an  aria...  ive  (linear, 
regression  model  seemingly  produces  a  stable  fit  to  synergistic  data. 

For  negative  intercorrelations,  on  the  other  hand,  the  results  reveal 

quite  a  different  picture.  In  both  panels,  there  is  a  sharp  "elbow"  in  the 

2 

top  curves  with  the  R  values  dropping  down  to  around  .20.  That  is,  the  fit 
of  a  linear  regression  model  to  synergistic  data  is  very  much  influenced 
by  the  degree  of  negative  intercorrelation.  (The  dip  in  the  curves  around  .00 
intercorrelation  in  the  right  panel  will  be  taken  up  later.) 

Figure  2  shows  the  r2  values  for  the  fit  of  a  multilinear  regression 
model  (Equation  8)  to  the  same  data.  The  two  panels,  representing  the  results 
for  25  and  100  stimulus  cases,  respectively,  are  also  quite  similar.  The 
top  curves  are  relatively  flat  for  positive  intercorrelations  with  an 
asymptote  of  around  .75.  For  negative  intercorrelations,  there  is  again  an 
elbow  in  the  R^  values.  Compared  to  Figure  1,  the  top  curves  are  consistently 
higher;  the  fit  to  the  adding  data,  however,  is  virtually  unchanged. 


Insert  Figure  2  about  here 


A  more  revealing  view  of  the  difference  between  Figures  1  and  2  can  be 

2 

obtained  by  computing  the  improvement  in  R  gained  from  a  multilinear  regression 
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model  over  an  additive  model.  These  AR^  values  are  shown  in  Figure  3 .  As 
before,  the  two  panels,  for  25  and  100  stimulus  cases,  are  fairly  similar. 

For  positive  intercorrelations,  the  AR^  values  stabilize  at  about  .05,  i.c., 
the  improvement  in  variance-accounted-f or  when  going  from  an  additive  to 
a  multilinear  model  is  almost  constant.  For  negative  intercorrelations, 
however,  the  improvement  in  fit  increases  dramatically  with  higher  negative 
values.  At  the  most  extreme,  the  AR^  values  approach  .25  when  r  *-.90  ir.  the 
right  panel. 


Insert  Figure  3  about  here 

o 

To  summarize,  the  R  results  show  that  a  linear  regresssion  model 
appears  to  do  reasonably  well  in  describing  synergistic  data  when  the  cues 
are  positively  correlated.  But,  when  the  "correct"  multilinear  regression 
model  is  used,  there  is  an  improvement  of  about  .05  in  the  r2  values.  However 
when  the  cues  are  negatively  correlated,  a  linear  regression  model  is 
definitely  inferior  to  a  multilinear  regression  model.  Moreover,  the 
larger  the  negative  intercorrelation,  the  greater  the  improvement  by  using 
the  "correct"  regression  model. 

Beyond  variance-accounted-for  measures,  the  most  frequently  used 
descriptive  indices  are  the  regression  weights.  Figure  4  shows  the 
standardized  regression  (Beta)  weights  for  the  crossproduct  terms  in  Equation 
8.  Except  for  a  falloff  with  highly  negative  cue  correlations,  the  Beta 
weights  are  relatively  sizable.  The  values  range  from  around  .65  for 
highly  negative  intercorrelations  in  the  left  panel  to  nearly  .90  for 
highly  positive  intercorrelations  in  the  right  panel. 

For  comparative  purposes,  the  Beta  weigh  ~  for  the  cross  product  of  a 
multilinear  model  fit  to  adding  data  are  shown  in  the  middle  of  the  figure. 
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As  expected,  these  values  are  uni  for.aly  near  zero.  'finis ,  tnt  ......  *>.a 

weights  provide  suggestive  indications  that  the  presence  of  a  syner,; . 
does  make  a  consistent  difference. 


Insert  Figure  4  about  here 


While  descriptive  indices  can  be  quite  useful  in  summarizing  data,  cr.ey 
are  of  course  inherently  incapable  of  saying  whether  a  syn.  'gisir.  is  prase.*., 
or  not.  That  is,  they  cannot  be  used  to  answer  yes-no  questions.  Therefore, 
such  indices  can  be  quite  deceptive  if,  say,  a  high  value  is  used  to 

support  a  linear  regression  model.  There  is  never  any  way  of  knowing  whether 
the  observed  R2  value  is  good,  bad,  or  mediocre.  For  a  more  complete  discussion 
of  this  issue,  see  Anderson  and  Shanteau  (1977;  also  see  Shanteau,  1977). 

The  appropriate  way  to  deal  with  such  issues  is  to  employ  inferential 
test  statistics.  Accordingly,  the  next  section  will  deal  with  various 
inferential  indices  designed  to  detect  the  presence  of  synergisms. 

Inferential  Indices 

Besides  providing  descriptive  information,  the  Beta  weights  for  the 
crossproduct  terms  can  also  be  tested  for  significance.  The  proportions 
of  significant  Beta  weights  (out  of  90  for  each  point)  are  plotted  in  Figure 
5.  For  100  stimulus  cases,  the  right  panel  shows  that  the  weights  are 
uniformly  significant.  In  contrast,  the  left  panel  for  25  cases  reveals 
that  only  highly  negative  intercorrelations  produced  proportions  near  one. 

For  positive  intercorrelations,  the  proportions  drop  to  near  .75.  Thus,  the 
results  for  regression  weights  reveal  that  they  are  generally  sensitive  to 
the  presence  of  a  synergism.  However,  this  sensitivity  is  reduced  when 
25  stimulus  cases  are  used  and  when  the  intercorrelations  are  positive. 
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Insert  Figure  5  about  here 


There  are  some  well-known  problems  with  using  regression  weights  to 
examine  the  importance  of  terms  such  as  crossproducts.  One  problem  is 
that  when  the  cues  arc  intercorrelated ,  the  order  in  which  the  terms  are 
examined  can  influence  the  magnitude  of  a  weight  (Darlington,  1969). 

Also,  the  scaling  of  the  stimulus  metric  can  influence  the  apparent  importance 
of  even  standardized  weights  (Anderson  &  Shanteau,  1977).  While  such  problems 
were  controlled  for  in  the  present  simulations,  these  shortcomings  would 
generally  make  regression  weights  impractical  for  testing  synergisms  in  real 
data. 

One  procedure  which  avoids  these  problems  is  the  hierarchical  test 
proposed  by  Cohen  and  Cohen  (1975).  Briefly,  this  procedure  involves  testing 
the  AR  presented  in  Figure  3  with  an  F-ratio.  As  can  be  seen  in  Figure  6, 
the  F  values  are  considerably  higher  for  the  100  stimulus  cases  in  the  right 
panel.  Moreover,  there  is  a  pronounced  decline  in  the  F  values  as  the 
intercorrelations  go  from  negative  to  positive. 


Insert  Figure  6  about  here 


When  these  F  values  are  tested  for  significance,  the  proportions  for  the 
100  stimulus  cases  in  Figure  7  are  uniformly  at  1.0.  However,  the  proportions 
for  25  cases  in  the  left  panel  range  between  .75  and  1.0.  In  addition, 
positive  intercorrelations  are  less  ]  ikely  to  produce  significant  results 
than  negative  intercorrelations. 
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Insert  Figure  7  about  here 


On  the  whole,  the  hierarchical  procedure  is  quite  sensitive  to  the 
presence  of  a  synergism,  especially  in  the  larger  stimulus  set.  In  the  smaller 
set,  there  is  a  slightly  higher  chance  of  detecting  synergisms  when 
cues  are  negatively  correlated.  By  and  large,  however,  the  hierarchical  r_.  t 
results  were  little  affected  by  any  of  the  present  environmental  manipulations. 

While  the  hierarchical  test  was  quite  good  at  detecting  synergisms ,  it 
is  not  capable  of  saying  whether  there  is  more  involved  than  simple  bilinearity. 
That  is,  showing  that  a  crossproduct  is  present  does  not  rule  out  the  presence 
of  other  more  complex  terms.  One  way  to  check  that  is  to  examine  the  iack- 
of-fit  after  the  additive  and  multiplicative  terms  have  been  extracted.  As 
described  by  Draper  and  Smith  (1966),  F-ratios  for  lack-of-fit  can  be  used 
to  test  the  unaccounted-for-variance.  The  average  F-ratios  are  shown  in 
Figure  8  for  the  25  and  100  stimulus  cases,  respectively.  The  curve 
shapes  are  for  the  most  part  similar  across  the  two  panels,  with  the  largest 
F-ratio  found  for  negative  intercorrelations. 


Insert  Figure  8  about  here 


The  proportions  of  F-ratio  that  were  significant  are  shown  in  Figure  9. 
While  the  proportions  are  considerably  higher  for  the  100  stimulus  cases, 
the  results  are  uniformly  greater  than  the  chance  levels  observed  for  the 
adding  model.  Of  course,  since  deviations  from  multilinearity  are  being 
tested,  and  since  the  synergistic  models  contain  no  such  deviations,  the 
proportions  should  all  be  at  the  chance  level .  It  would  thus  appear  that 
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this  procedure  lends  to  an  Inflated  type  I  error  rate.  Based  'r'  pre.-.e.-.; 

results,  the  lnck-of-fit  test  is  apparently  overly  sensitive  to  nonextster. . 
deviations  and  therefore  should  not  be  used. 


Insert  Figure  9  about  here 


Post-Hoc  Analyses 

Aside  from  the  planned  analyses  reported  to  this  poin.  ,  several  additional 
analyses  were  performed  based  on  some  unanticipated  results.  These  involved 
the  surprising  influence  of  some  atypical  stimulus  sets  and  the  anomalous 
findings  observed  with  zero-intercorrelation  sets. 

Between- set  differences.  In  the  results  presented  so  far,  the  findings 
have  been  averaged  over  the  nine  stimulus  sets  constructed  for  each  intercor¬ 
relation  condition.  Since  particular  efforts  were  made  to  ensure  the 
comparability  of  these  sets,  there  was  little  reason  to  expect  any  sizable 
differences  between  them.  Nevertheless,  there  were  some  notable  between- 
set  differences  in  the  values  for  various  indices. 

One  of  the  most  striking  examples  is  summarized  in  Table  3;  the  results 
were  taken  from  the  .00  intercorrelation/100-stimulus-case  condition  with  the 
data  generated  by  the  adding-multiplying  model  (Equation  4).  The  row 
entries  present  the  results  for  each  of  the  nine  constructed  stimulus  seta. 

In  the  first  column,  the  observed  cue  intercorrelation  values  can  be  seen 
to  be  extremely  close  to  the  prespecified  .00  value.  Although,  not  shown, 
the  means,  standard  deviations,  and  distributions  were  also  quite  close  to 
their  prespecified  values  (see  Table  2  for  an  example).  The  fit  of  the 
linear  regression  model  (Equation  7)  led  to  average  values  in  the 


second  column  which  range  from  .61  to  .66 — with  one  notable  exception. 
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The  fifth  stimulus  set  lias  an  R“  value  of  only  .37.  A  similar  discrepancy 
can  be  observed  in  the  third  column  for  the  values  from  the  fit  of  a 

multilinear  model  (Equation  8). 


Insert  Table  3  about  here 


The  average  Beta  weights  for  the  crossproduct  tenr.  are  given  in  col„r.r. 
four  with  the  number  significant  (out  of  10)  in  parentheses.  The  aw:.... 

weight  falls  between  .77  and  .88,  again  with  the  exception  of  the  fifth 

set  which  has  a  value  of  .66.  Even  larger  differences  can  be  seen  for 
the  average  hierarchical  F-ratios  in  column  five  and  the  lack-of-fit 
F-ratios  in  column  six.  In  both  cases,  the  results  for  the  fifth  stimulus 

set  are  far  out  of  line  from  the  other  eight  stimulus  sets. 

A  more  typical  set  of  data  is  presented  in  Table  4  for  -.90  correlation / 
100-stimulus-case  condition.  For  these  results,  the  eighth  stimulus  set, 
and  to  a  lesser  extent  the  fourth  set,  stand  apart.  As  an  example,  the 
values  for  the  multilinear  regression  model  range  from  .45  to  .60  with 
the  exception  of  .30  for  the  eighth  set  and  .34  for  the  fourth.  Most  of 
the  other  conditions,  although  not  shown  to  save  space,  produced  results 
similar  to  Table  4  in  that  one  or  two  of  the  stimulus  sets  stood  out  from 
the  others. 


Insert  Table  4  about  here 


In  an  effort  to  localize  the  source  of  these  discrepancies,  the  means 
and  standard  deviations  of  the  simulated  data  were  computed.  As  can  be  seen 
from  the  averages  reported  in  column  seven  ol  Tables  3  and  4,  the  means  are 
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quite  similar  across  all  the  stimulus  sets.  However,  the  stance _  dwvi.it.  ' .... 

in  column  eight  are  another  matter.  The  values  are  markedly  smaller  f ur 
the  fifth  set  in  Table  3  and  the  eighth  set  in  Table  4.  That  is,  the  rang.., 
variability  of  the  data  generated  for  these  sets  is  compressed  relative  to 
the  other  stimulus  sets.  This  in  turn  apparently  produced  smaller  values 
for  various  statistical  indices.  In  short,  it  was  more  difficult  to  detect 
the  presence  of  a  synergism  in  the  data  generated  from  these  stimulus  . 

Obviously,  this  would  make  the  generalizability  of  the  rest. is  obtainea  fro., 
these  sets  highly  suspect. 

Zero-correlation  results.  In  several  of  the  figures,  the  .00  inter- 
correlation/100-stimulus-case  condition  appears  to  have  produced  anomalous 
results.  For  instance,  in  Figures  1  and  2  the  top  curves  in  the  right  panels 
show  a  marked  dip  for  the  .00  intercorrelation  value.  While  the  reason 
for  this  anomaly  is  not  entirely  clear,  .00  intercorrelated  stimulus  sets 
have  also  been  observed  to  be  unusual  in  other  studies  (Stewart,  1980). 

One  contributing  factor  may  be  the  relative  homogeneity  of  the  stimulus 
sets  with  .00  intercorrelation.  In  the  first  column  of  Table  3,  for  instance 
the  observed  intercorrelation  values  are  all  extremely  close  to  .00. 

Such  close  similarity  was  not  observed  for  any  of  the  other  intercorrelation 
conditions.  In  Table  4,  for  example,  the  observed  intercorrelation  values 
range  from  -.85  to  -.93.  It  would  thus  appear  in  the  .00  intercorrelation 
condition,  the  stimulus  sets  were  much  closer  to  criterion  value  than  was 
the  case  elsewhere.  It  would  therefore  appear  that  the  greater  homogeneity 
of  the  .00  intercorrelation  sets  apparently  accentuated  any  differences  from 
the  other  conditions. 

Since  the  stimulus  sets  in  the  .00  intercorrelation  are  so  similar,  this 
emphasizes  all  the  more  the  uniqueness  of  th  •  fifth  set  in  Table  3.  That  is 
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of  all  the  inter correlation  conditions,  this  would  seer  to  be  the  least 
likely  to  produce  an  anomalous  stimulus  set.  In  fact.,  ^ he  fifth  set, 
with  a  value  of  .01 ,  was  the  only  one  not  to  have  an  intercorrelation 
value  of  .00.  It  appears  that  even  this  very  slight  deviation  in  the  fifth 
set  may  have  contributed  to  the  anomalous  results.  This  finding  suggests 
that  atypical  stimulus  sets  can  occur  even  in  the  most  unexpected  and  - : . 
controlled  situations. 

Discussion 

There  are  three  noteworthy  findings  in  the  present  study.  First, 
multiplicative  synergisms  do  make  a  major  difference  in  nonorthogonal 
designs.  When  trying  to  account  for  synergistic  data,  it  matters  a  great 
deal  whether  a  "correct"  or  an  "incorrect"  regression  model  is  used.  While 
the  effect  is  more  pronounced  for  negative  intercorrelations,  the  impact 
of  a  synergism  can  be  seen  throughout  the  range  of  cue  intercorrelations. 
Thus,  investigators  who  continue  to  ignore  the  possibility  of  synergisms 
may  be  overlooking  some  potentially  very  important  relationships. 

Second,  much  of  what  might  be  considered  common  practice  in  the  analysi 
of  nonorthogonal  data  is  called  into  question  by  the  present  results.  For 
instance,  some  of  the  indices  regularly  used  to  analyze  judgmental  data, 
e.g.,  R2,  were  found  to  be  insensitive  to  the  presence  of  synergisms.  On 
the  other  hand,  there  were  some  less  common  measures  which  were  sensitive  to 
synergisms,  e.g.,  AR^,  and  which  could  be  routinely  incorporated  into  j uu_,mo 
analyses.  In  addition,  a  rather  surprising  result  was  that  some  stimulus 
cue  sets  were  better  than  others  at  revealing  the  presence  of  synergisms.  s 
all  sets  had  to  meet  some  rather  stringent  qualification  requirements,  this 
suggests  that  there  may  be  some  unappreciated  difficulties  in  generalizing 
results  obtained  from  nonorthogonal  designs. 


i 
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Final  Ly,  tin-  success  ot  Cue  -ireti-nt  simulat.on  an, ...... 

on  several  grounds.  On  the  one  hand,  the  approach  proved  quite  useful  lr. 
investigating  some  issues  which  would  have  been  difficult,  it  not  imp os.  ci.. 
to  study  empirically.  On  the  other  hand,  several  new  research  issue.-.,  were 
raised  which  can  be  addressed  in  empirical  investigations.  For  instance, 
the  rote  of  error  in  judgmental  data  might  become  of  special  concern  in 

future  analyses.  Although  not  without  limitations,  the  simultaneous  ii;; _ - 

tion  of  environmental  conditions,  subject  behavior,  and  re*,  n..a-  . 

can  provide  a  fruitful  basis  for  investigating  many  other  issues.  The 
implications  of  each  of  these  three  findings  will  be  taker,  up  in  t he  rtt.ut-L 
of  the  Discussion. 

The  Impact  of  Synergisms 

The  present  results  are  quite  clear  in  showing  that  the  presence  of  a 
synergism  does  make  a  difference.  Regardless  of  the  environmental  condition- 
investigated,  there  was  always  an  improvement  in  the  fit  of  a  multilinear 
regression  model  over  a  linear  model  for  synergistic  data.  While  the  size 
of  the  improvement  did  vary  depending  on  which  indices  and  conditions  were 
used,  there  was  not  a  single  occasional  in  any  of  the  present  simulations 
where  an  improvement  failed  to  appear. 

Cue  intercorrelations .  The  pervasiveness  of  the  influence  of  synergisms 
was  somewhat  unexpected.  Perhaps  most  surprising  was  the  persistent  effect 
found  for  the  high  positive  intercorrelation  conditions.  When  cues  are  cl'.-., 
bound  together,  e.g.,  by  a  correlation  of  +.90,  the  fit  might  be  expeetcc.  to 
be  insensitive  to  the  form  of  the  regression  model.  That  is,  for  highly 
correlated  cues,  high  values  will  follow  from  high  cues  and  low  values  wi , 
follow  from  low  cues  regardless  of  whether  a  linear  or  a  multilinear  model  >s 


used.  Moreover,  the  degree  of  insensitivity  might  be  expected  to  increase  as 
the  correlation  approaches  unity. 
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Instead,  the  present  results  revealed  ar.  almost  constant  d:  <"fe~ence 
between  the  fit  of  linear  and  mul tilinear  regression  models  lor  positive 
intercorrelations.  It  would  appear  that  the  degree  of  intercom' letio 

when  it  is  positive,  is  of  little  relevance.  Thus,  up  to  the  limits  of 
the  present  simulation  analyses,  the  influence  of  a  synergism  appears  to 
be  quite  consistent  for  positive  cue  correlations. 

Some  rather  curious  results  appeared  for  the  zero-intercorre^otion 
condition.  Basically,  the  results  are  quite  similar  to  thes  observed  for 
positive  correlations.  However,  several  of  the  figures  revealed  "dips" 
and  other  discontinuities  for  zero  intercorrelations.  While  the  source 
this  irregularity  is  not  entirely  understood,  the  important  result  is  neve 
theless  unchanged:  synergisms  have  just  as  much  effect  of  this  intercorre 
tion  condition  as  in  any  positive  condition. 

A  rather  different  picture  emerges  for  negative  cue  intercorrelations 
The  greater  the  size  of  the  negative  intercorrelation,  the  greater  the 
disparity  between  the  fit  of  linear  and  multilinear  models  to  synergistic 
data.  That  is,  the  size  of  the  negative  correlation  influenced  how  much 
was  lost  by  ignoring  the  presence  of  a  synergism. 

While  some  difference  between  positive  and  negative  intercorrelations 
had  been  anticipated,  the  magnitude  of  the  difference  was  not.  With  high 
negative  correlations,  the  discrepancy  between  the  fit  of  linear  and 
multilinear  models  approaches,  and  in  some  cases  even  exceeds,  the  fit  of 
the  linear  model.  That  is,  the  variance-accounted-for  can  actually  be 
doubled  by  shifting  from  a  linear  to  a  multilinear  regression  model.  Bec- 
of  the  disparity  between  these  results  and  the  general  acceptance  of  lines 
models,  the  role  of  such  models  will  next  be  considered  in  some  depth. 

Linear  models  in  decision  making .  A  great  deal  of  knowledge  has  been 
accumulated  about  the  use  of  linear  models  in  summarizing  human  judgments. 
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Much  of  what  has  been  found  surest.-,  that  neither  correct  mod  ’  *.  r  nor 
correct  weights  are  important  for  getting  an  adequate  desc.r .pcion  oi  ;:„v 
data  (Slovic,  i’ischhoff,  &  Lichtenstein,  1977).  The  present  resuiLs  for 
synergistic  data  would  imply  the  need  for  some  modifications  in  this  view 
(The  issue  of  weights  will  be  taken  up  separately  below.) 

There  are  a  number  of  papers  in  the  Literature  which  LaVe  ... 

linear  models  can  do  a  good  job  of  describing  nonadditivc  data 
Yntema  &  Torgerson,  1961).  In  perhaps  the  best  known  of  '-.e.je  paper* , 
Dawes  and  Corrigan  (1974,  p.  98)  concludes  that  "linear  monels  are  gooe 


approximations  to  all  multivariate  models  that  are  conditionally  mono. r... 


in  each  predictor  variable."  The  authors  go  on  to  add  that  the  linear 
approximations  improve  with  increasing  error.  Hence,  there  has  b_en  a 
widespread  feeling  that  distinguishing  model  form  is  unimportant  for  most 
multiple  regression  analyses  of  judgmental  data.  Indeed,  many  of  the 
approaches  which  use  regression  procedures,  such  as  the  hen's  model 
approach  (Hammond,  Stewart,  Brehmer,  &  Steinmann,  1975),  routinely  ignore 
anything  but  a  linear  regression  model. 

In  contrast,  Anderson  and  Shanteau  (1977)  cautioned  against  the  rout 
use  of  linear  models.  Among  other  shortcomings,  they  offe  d  an  example 
of  multiplying  data  which  satisfies  conditional  monotonicity  but  which  is 
clearly  nonlinear.  While  a  best-fitting  linear  model  correlated  .885  wit 
the  data,  the  fit  was  far  from  adequate.  On  a  100-point  scale,  the  line, 
model  was  providing  estimates  which  ranged  from  -25  to  -t-25  for  a  c^ts  v... 
of  0.  The  basic  problem  was  that  the  data  displayed  a  diverging  pattern 


lines  and  a  linear  model  can  only  produce  parallel  lines.  In  short, 
model  was  not  adequate  to  describe  multiplying  data. 
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And ur. son  and  Shaniuau  (3977)  wen:  on  to  point  out  that  iir.,-..r  ...  .el:, 
can  be  useful  in  applications  involving  data  prediction.  When  the  go..i  is 
to  understand  psycho1. op, i ca  1  processes,  however,  linear  models  can  be 
deceptive.  For  instance,  synergistic  processes  can  be  easily  misinterpreto,. 
as  being  additive  if  linear  models  are  used  exclusively  in  data  analysts 
(for  an  actual  example,  see  Shanteau,  1977). 

The  present  findings  extend  the  arguments  against  the  routine  use  c. 
linear  models  in  two  ways.  First,  even  the  predictive  use  c  '  lineal  modi  1 ; 
can  be  questioned  when,  at  best,  the  loss  in  variance-accountec-f or  is  3  . 

At  worst,  a  linear  model  can  account  for  only  half  of  the  systematic  verge 
Moreover,  in  all  cases,  the  loss  in  variance-accounted-for  was  significant 
when  a  linear  model  was  used.  Of  course,  whether  losses  of  this  magnitude 
are  within  the  realm  of  a  "good  approximation"  might  still  be  subject  to  some 
debate.  Nevertheless,  the  present  findings  suggest  that  the  predictive 
ability  of  linear  models  should  not  be  uncritically  accepted. 

Second,  previous  evidence  against  linear  models,  such  as  that  offered 
by  Anderson  and  Shanteau,  has  been  primarily  based  on  analyses  of  variance 
applied  to  orthogonal  designs.  The  results  offered  here  demonstrate  that 
synergisms  also  make  an  important  difference  in  regression-based  analyses 
of  nonorthogonal  designs.  Thus,  neither  the  type  of  statistical  analyses 
nor  the  experimental  design  are  relevant  to  the  argument  that  linear  models 
can  be  misleading  when  applied  to  synergistic  data. 

Weights  : or  linear  models.  Several  recent  papers  have  pointed  out  that 
previous  arguments  about  the  insens i 1 i vi ty  ;  or  weights  for  linear  models 
may  be  inappropriate.  Previously,  investigators  such  as  Wainer  (1976),  as 
we1!  as  Dawes  and  Corrigan  (1974),  had  argued  that  equal  weights  (or  even 
rarlom  weights  in  some  circumstances)  can  do  out  as  well  as  optimal  weights 
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in  linear  model However ,  Newman  (  :  9  7  7 )  demonstrated  taut ,  wr.xle  equal 
weights  can  provide  good,  approximations  when  cues  are  positively  correlates, 
equal  weights  are  generally  inferior  when  cues  are  negatively  correlated. 
Thus,  previous  arguments  about  equal  weights  in  linear  models  apparently 
cannot  be  generalized  to  conditions  of  negatively-correlated  cues  (also 
see  John  &  Edwards,  1978). 

The  results  here  expand  and  complement  the  findings  of  Newman  (197’). 
While  Newman  was  concerned  with  showing  that  weights  in  el  X  —  Hi  iS  Q  T  DiO  Wi  d  1 
matter  for  negative  intercorrelatior.s ,  the  present  results  show  that 
form  also  matters.  Thus,  when  cues  are  negatively  correlated,  neither 
the  weights  nor  the  form  of  the  model  should  be  taken  for  granted. 

Positive-vs-negative  in ter cor re  1 ations .  Since  negatively  intercorrela¬ 
tions  produce  such  different  results,  it  is  appropriate  to  ask  about  the 
conditions  under  which  such  intercorrelations  might  be  observed.  Perhaps, 
negatively  correlated  cues  are  relatively  rare  in  reality  and  so  would  not 
be  much  cause  for  concern.  After  all,  perceptual  judgments  for  instance 
are  made  in  the  context  of  numerous  largely  redundant,  i.e.  ,  positively 
correlated,  cues  (Brunswik,  1956;  also  see  Hammond,  1981). 

However,  negative  correlations  may  in  fact  be  the  rule  not  the 
exception  in  decision  making.  Many  decision  problems  are  only  problems 
because  the  cues  are  inversely  related.  For  example,  selecting  a  new  car 
would  be  trivial  if  the  attributes  were  positively  correlated,  i.e.,  if  the 
cheapest  car  was  also  the  best  looking.  In  reality,  however,  such  a  car 
does  not  exist  and  instead  we  are  forced  to  make  tradeoffs  between  conflict, 
attributes.  Thus,  negative  intercorrelations  will  be  found  precisely  in 
those  situations  where  decisions  are  most  likely  in  reality. 

A  more  elegant  discussion  of  the  role  o.  negative  intercorrelations  can 
be  found  in  McClelland  (in  press).  He  shows  that  if  the  set  of  alternatives 


Detection  of  Synergisms 
27 

is  restricted  to  those  that  are  nondominated ,  i.e. ,  on  the  Par  ... 
frontier,  then  the  cues  will  necessarily  be  negatively  correlated.  Thai 
only  by  including  dominated  alternatives  can  a  set  have  anything  but 
negatively  correlateu  cues.  Therefore,  excluding  inferior  alternatives 
produces  negative  intercorrelations. 

Moreover,  decision  situations  involving  negative  intercorrelatio.'..- 
also  quite  likely  to  lead  to  synergistic  processing  rules.  For  ins , 
in  risky  decision  making,  payoffs  and  probabilities  are  get  : rally  ue/nv... 
correlated,  i.e.,  high  payoffs  have  low  probabilities  and  low  payoffs  hn.'e 
high  probabilities.  Further,  both  the  optimal  decision  rule  (Eawards,  . 
and  the  strategy  generally  used  by  subjects  to  make  risky  decisions  (Sn.t.. 
1975)  involve  the  multiplication  of  probability  and  payoff.  Similarly,  utir 
synergistic  rules  are  likely  to  be  found  in  precisely  those  conditions  w;v.\ 
negative  intercorrelations  are  observed  (Hammond,  1981). 

Another  way  of  looking  at  positive  versus  negative  correlations  is  m 
terms  of  the  assumed  resources  availaole.  ’In  a  world  involving  positively- 
correlated  attributes,  there  is  no  implied  limit  on  resources,  i.e.,  it  is 
theoretically  possible  to  get  the  most  on  all  attributes  at  the  same  time. 
Thus,  positive  intercorrelations  suggest  an  unlimited-resources  view  of  the 
world.  In  contrast,  negatively-correlated  attributes  imply  a  limit  on 
available  resources,  i.e.,  it’s  not  possibl  e  to  simultaneously  get  the  most  on 
every  attribute.  This  latter  view  may  be  much  more  reasonable  in  a  worli. 
that,  in  fact,  requires  choices  involving  limited  resources. 

Statistical  Issues  and  Synergisms 

For  purposes  of  detecting  synergisms,  two  findings  at  a  statistical  level 
stand  out.  The  first  concerns  the  comparison  of  various  statistical  inuiees. 


The  second  relates  to  experimental  design  and  the  influence  of  atypical  stimulus 


sets. 
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The  implications  of  each  of  these  lendings  will  be  dise..„.,uc.  .r. 
section. 

Comparison  of  statistical  indices.  As  can  be  seen  in  the  figure-  ,  the 
various  indices  were  all  sensitive  to  some  extent  to  the  presence  of  a 
synergism.  Regardless  of  the  index,  the  values  were  uniformly  higher  wrier, 
the  "correct"  regression  model  was  applied  to  synergistic  data.  This  i  ... 
true  across  all  intercorrelation  values  and  across  the  two  stimulus  be. 
sizes . 

In  practice,  however,  several  of  the  indices  in  common  use  may  provide 
uncertain  information  about  the  presence  of  a  synergism.  For  instance,  a 
R2  value  of  .70  for  the  fit  of  a  linear  model  may  look,  good — until  it  io 

O 

discovered  that  a  multilinear  model  leads  to  a  R  value  of  .75  for  the 
same  data  (these  values  are  taken  from  Figures  1  and  2).  The  difficulty 
with  indices  such  as  R^  (and  other  corre.lation-based  measures)  is  that 
it  is  impossible  to  know  by  looking  at  a  single  value  whether  the  fit  is 
good  or  not.  Only  by  comparing  the  fits  for  various  models  can  any  evaluative 
statements  be  made. 

Unfortunately,  comparative  analyses  using  alternative  models  are  seldom 
performed.  Worse  yet,  there  is  no  limit  to  the  number  or  variety  of  alternative 
models  that  might  be  considered.  In  short,  measures  such  as  R  do  not 
provide  an  adequate  basis  for  detecting  synergisms  (also  see  Shanteau,  1977). 

Other  common  descriptive  measures,  such  as  the  size  of  the  regression 
weight  for  the  crossproduct  term,  are  also  inadequate.  The  problem  is 
that  when  cues  are  intercorrelated ,  the  order  in  which  the  analysis  is 
conducted  can  influence  the  size  of  the  weights  (Darlington,  196S) .  This 
means  that  the  size  of  the  weights  depends  on  a  variable  that  is  under  direct 
control  of  the  investigator.  Thus,  regression  weights  provide  an  uncertain 
indicator  of  the  presence  of  a  synergism. 
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The  only  lioseripi  ive  measure  that  can  be  recommended  on  r'..  b. 
of  the  simulations  here  is  AR  .  Thi.^  measure,  obtained  from  the  difference 
in  fit  of  linear  and  multilinear  models,  was  generally  sensitive  to  the 

presence  of  a  synergism.  In  addition,  the  hierarchical  test  discussed 
below  provides  a  test  of  significance  that  is  related  to  this  measure. 
Therefore,  AR-  would  be  the  preferred  descriptive  index. 

The  present  results,  therefore,  show  that  some  widely  used  measure^  in 
regression  analyses  are  not  suitable  for  detecting  synergi.  s.  This 
well  explain  why  synergisms  have  been  so  infrequently  reported  in  previous 
analyses  of  nonorthogonal  designs.  In  contrast,  numerous  instances  of 
synergisms  have  been  found  in  studies  involving  orthogonal  designs  (e.g. , 
Shanteau,  1981). 

Of  the  inferential  indices  considered  here,  the  measure  of  choice 
appears  to  be  the  hierarchical  test  proposed  by  Cohen  and  Cohen  (1975).  The 
test  correctly  detected  the  presence  of  synergisms  in  all  the  100  stimulus- 
case  conditions.  A  slight  decrement  in  detectability  rates  was  observed  for 
the  25  stimulus-case  conditions.  But  even  at  its  worst,  the  hierarchical 
procedure  detected  75%  of  the  synergisms  (see  Figure  7).  Equally  important, 
the  false-alarm  rates  for  the  hierarchical  tests  were  consistently  below 
10%.  In  short,  the  hierarchical  test  was  quite  sensitive  to  the  presence  of 
synergisms. 

The  hierarchical  procedure  has  been  advocated  (Arnold  &  Evans,  1979) 
precisely  because  of  its  potential  for  evaluating  multiplicative  component u. 

A  problem  in  previous  regression  ana]yses  has  been  how  to  analyze  multilinear 
models  when  the  independent  variables  are  not  measured  on  a  ratio  scale. 

In  those  cases,  an  additive  component  is  introduced  into  the  regression  model 
and  direct  tests  of  the  crossproduct  may  not  he  revealing.  However,  as 


Detection  of  Synergisms 
30 

shown  hero ,  the  presosu-o  of  an  additive  ter.u  has  little  influent"  cr,  the 
hierarchical  test  results  (see  Figures  6  and  7).  Thus,  the  present  recom- 

o 

mendetion  would  be  to  include  routinely  AR-  and  the  hierarchical  procedure 
in  multiple-regression  analyses  of  judgmental  data  (also  see  Stahl  &  Harrell, 
1981). 

In  contrast,  the  iack-of-fit  test  is  notable  because  of  its  poor 
performance.  The  major  problem  appears  to  be  an  overly  large  sensitivity  to 
nonexistent  deviations  from  the  multilinear  model.  Of  cour_  -.it's  possible 
that  the  inflated  detection  rate  may  be  due  to  some  facet  of  the  present 
simulation  analyses.  That  is,  would  real  data  produce  a  better-behaved  test 
statistic? 

To  examine  this  question  further,  it  is  worth  considering  the  study  by 
Shanteau  and  Nagy  (1979).  They  applied  a  comparable  lack-of-fit  procedure 
to  test  the  adequacy  of  a  multilinear  model  for  dating  decisions.  The  model 
described  the  data  quite  well  and  was  highly  accurate  in  predicting  actual 
dating  choices.  Despite  the  apparent  good  fit  it,  however,  the  lack-of-fit  tests 
revealed  significant  discrepancies  for  over  two-thirds  of  the  subjects. 

Additional  analyses  revealed  no  discernable  locus  to  the  discrepancies,  and 
the  deviations  seemed  to  be  quite  small.  It  thus  appears  that  the  test  is 
overly  sensitive  to  small  deviations  in  real  data.  Taken  together  with  the 
simulation  results,  it  would  appear  that  until  more  is  known  about  the  properties 
of  the  test,  it  cannot  be  recommended  for  regular  use. 

Idiosyncrasies  in  nonorthogonal  designs.  Perhaps  the  most  unexpectec 
finding  to  come  out  of  the  simulations  was  the  occasional  occurence  of  an 
atypical  stimulus  set.  What  made  this  so  surprising  was  that  all  stimulus 
sets  had  to  pass  a  number  of  stringent  qualification  tests  before  being  accepted. 
The  goal  was  to  produce  stimulus  sets  that  were  as  homogeneous  as  possible. 
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Yet,  one  or  two  atypical  sets  were  found  in  nearly  every  cor, .  Arr.o.'.. 

other  differences,  such  sets  led  co  nonconforming  results  concerned  the 
sensitivity  of  various  indices  to  the  presence  of  a  synergism.  What  this 
means  is  that  the  ability  to  detect  a  synergism  depends  on  the  particular 
stimulus  set  selected. 

Of  course,  it  is  possible  that  even  more  stringent  qualification  te.-e... 
would  have  led  to  more  homogeneous  sets.  Indeed,  based  on  hindsight,  .-.uny 
of  the  present  atypical  sets  could  have  been  eliminated  u_  cheeking 
variability  of  the  expected  (pre-error)  response  values.  However,  dern  r..  .r.g 
stricter  selection  criteria  would  not  be  feasible  on  two  grounds.  First, 
at  a  practical  level,  nonorthogonal  designs  are  frequently  used  in  research 
settings  which  have  little  or  no  flexibility.  For  instance,  a  marketing 
researcher  has  little  if  any  control  over  the  product  alternative  set. 

In  such  settings,  the  researcher  may  have  no  choice  but  to  use  the  available 
stimulus  cases. 

Second,  at  a  theoretical  level,  unless  the  stimulus  sets  are  identical, 
they  can  never  be  homogeneous  for  all  purposes.  While  it  might  be  possible 
to  construct  qualification  tests  to  insure  that  the  stimulus  sets  are 
equivalent  in  regard  to  detecting  synergisms ,  the  sets  might  still  be 
dissimilar  for  other  purposes.  That  is,  there  is  no  way  to  select  stimulus 
sets  that  are  homogeneous  for  all  applications.  Moreover,  since  many 
analyses  are  impossible  to  anticipate,  there  is  no  way  to  preselect  stimulus 
sets.  In  short,  it  is  not  feasible  to  develop  a  general-purpose  qualii 
procedure. 

The  inability  to  develop  general  preselection  criteria  means  that  a 
researcher  can  never  be  certain  whether  a  particular  stimulus  set  is  atypical 
or  not.  As  the  present  study  demonstrates,  even  using  as  many  as  100  stimulus 
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cases  is  no  protect  ion  (see  'i'.ibie  3).  This  means  that  the  rc-ui  t.->  .  .  ... 
even  fairly  large  <los  may  not  general  ire .  I'nfor : i.nn  tc ’. y  ,  the  .ty 

of  atypical  stimulus  set:;  raises  questions  about  the  gencr-ilizabilit)  o: 
many  previous  results  obtained  using  nonorthogonal  designs. 

There  are  at  least  three  options  for  alleviating  the  problem  of  l..ck  of 
generality.  The  first  option  is  to  replicate  all  findings  using  difie.v..-.- 
nonorthogonal  designs.  This  would  greatly  minimize,  but  not  elimir.  tc,  im¬ 
possibility  that  the  observed  results  will  fail  tc  replicatu  because  of  a  . 
atypical  design.  While  such  experimental  replications  are  of  course 
desirable,  they  may  be  impractical  in  many  settings. 

Second,  there  are  many  investigations  in  which  orthogonal  (factorial) 
designs  might  be  used  instead  of  nonorthogonal  designs.  Factorial  designs 
avoid  almost  all  of  the  problems  discussed  above  for  nonorthogonal  stimulus 
sets.  Specifically,  factorial  designs  lead  to  optimal  parameter  estimates 
and  model  tests.  While  factorials  do  have  shortcomings  for  judgment  research, 
the  disadvantages  have  frequently  been  overemphasized  relative  to  the 
advantages . ^ 

The  final  option  is  to  pretest  the  nonorthogonal  design  using  the  types 
of  simulation  analyses  performed  here.  That  is,  the  anticipated  behavioral 
models,  along  with  the  method(s)  of  analysis,  can  be  simulated  in  advance. 

In  this  way,  the  suitability  of  the  stimulus  design  for  answering  the  research 
question(s)  can  be  established  a  priori. 

The  Role  of  Simulation  Analyses 

There  are  both  important  advantages  and  important  disadvantages  to  the 
use  of  simulation  analyses  to  study  issues  such  as  synergisms.  On  the  positive 
side,  simulations  allow  the  study  of  psychological  problems  that  would  be 
intractable  using  an  empirical  approach.  On  _,.e  negative  side,  any  simulation 
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is  only  as  good  as  its  assumptions  and  some  of  the  assumption  ~....ue  cs..'. 

certainly  be  questioned.  Before  considering  these  pluses  and  minuses  in 
detail,  there  are  some  important  distinctions  that  need  emphasis. 

Simulations  as  a  psychological  research  tool  are  hardly  unique.  There 
are  numerous  applications  in  the  literature  of  useful  simulation  analyses. 

For  instance,  computer  models  of  subject  behavior  have  been  frequently  t;:...../yc c. 
to  analyze  cognitive  processes  in  problem  solving  behavior  (e.g..  Newel* 

Simon,  1963).  Similarly,  Monte  Carlo  simulations  have  lot  been  used  to 
analyze  the  properties  of  various  statistical  procedures  (e.g.,  Lincquist, 

1953,  pp.  78-90). 

What  separates  the  present  approach  from  these  earlier  simulation 
analyses  is  the  effort  here  to  simulate  all  stages  of  an  experiment. 

As  outlined  in  Table  1,  the  three-part  approach  involves  separate  simulations 
of  environment,  behavior,  and  analyses.  While  prior  approaches  have  concen¬ 
trated  on  simulations  of  behavior  or  analyses,  the  present  approach  is  to 
view  the  research  process  as  a  whole.  Therefore,  these  three  elements 

are  all  included  in  what  might  be  termed  an  experiment-simulation.  The 
advantages  and  disadvantages  of  this  approach  will  now  be  considered. 

Advantages.  There  are  at  least  four  advantages  to  experiment  simulations. 
The  first  is  that  it  is  possible  to  address  research  questions  that  could  not 
be  practically  investigated  in  any  other  way.  The  present  study,  for 
instance,  involves  1,620  separate  conditions.  To  run  even  one  subject  ir.  each 
of  these  conditions  would  clearly  be  prohibitive.  Thus,  studies  that  would 
be  impossible  to  conduct  empirically  can  still  be  approximated  through  experi¬ 
ment  simulations. 

A  second  advantage  is  that,  with  the  present  approach,  the  "truth"  is 


always  known.  That  is,  the  true  state  of  the  environment,  the  true  behavioral 
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model,  and  the  true  analytic  answer  are  always  known.  Know!  t 

"trutii.-,"  allows  a  number  of  analyses  to  be  performed  that  caul  a  not  be 
conducted  otherwise.  For  instance,  various  designs,  models,  and  analytic 
techniques  can  be  directly  compared  because  the  correct  answers  are 
known.  Therefore,  the  researcher  is  in  the  highly  env  table  position  a; 
knowing  the  truth  at  every  stage. 

The  third  advantage  is  that  experiment  simulations  can  be  esoc  to 
generate  more 'precise  empirical  investigations.  That  is,  sinaiutiOr.;. 
point  out  research  issues  and  areas  where  empirical  research  is  lackir.f  . 

For  instance,  some  marked  differences  between  positive  and  negative  ir.cerc.or- 
relations  conditions  were  demonstrated  in  the  present  simulations.  However, 
there  is  as  yet  relatively  little  empirical  evidence  to  demonstrate  how 
subjects  respond  under  the  range  of  conditions  studied  here.  Its  not  even 
clear  whether  subejets  would  use  synergistic  rules  under  all  intercorrelation 
conditions.  Empirical  research  is  clearly  necessary  to  answer  such  questions 
generated  by  the  experiment  simulations. 

The  final  advantage  is  that  experiment  simulations  can  be  employed 
to  investigate  entirely  new  research  issues.  Problems  which  not  have  yet 
been  considered  may  be  highlighted  by  performing  a  simulation.  In  the 
present  study,  for  example,  the  location  of  the  error  term  in  the  subject’s 
model  became  a  major  issue.  Yet,  because  no  prior  research  was  available, 
this  issue  could  not  be  addressed  empirically.  However,  synergisms  provide 
a  unique  opportunity  to  separate  stimulus  error  from  resonse  error  ana  co 
investigate  the  relative  magnitude  of  each.  Thus,  experiment  simulation^ 
have  helped  to  focus  attention  on  a  new  and  potentially  quite  interesting 
research  issue. 
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Pi  sadvant.'uu  ■  There  are  two  noteworthy  probi  in  the  (  ...u 

simulation  approach  used  here.  The  first  is  that  there  is  no  criterion  for 
whether  the  simulations  are  successful  or  not.  While  the  results  produced 
look  reasonable,  that  is  no  assurance  that  there  are  not  importantdifficult.es. 
As  a  check,  what  is  needed  is  to  compare  some  of  the  present  findings 
against  specific  empirical  results.  Until  such  checks  have  been  performed  , 
there  is  always  the  possibility  that  the  present  simulations  may  be  ur.rei^Ce.. 
to  empirical  reality. 

A  second  disadvantage  is  that  this  or  any  simulation  is  only 

as  good  as  its  assumptions.  If  the  assumptions  are  faulty,  then  so  necessarily 
will  be  the  results  from  the  simulation.  In  the  present  three-stage  approach, 
there  would  seem  to  be  relatively  little  cause  for  concern  in  regard  to  the 
simulations  of  the  environment  and  the  experimenter.  The  assumptions  made 
for  these  stages  were  largely  noncontroversial  and  in  line  with  standare 
research  procedures. 

The  status  of  the  subject  simulation  is  not  as  clear,  however,  since 
several  rather  arbitrary  assumptions  had  to  be  made.  Most  notably,  the  way 
in  which  error  was  incorporated  may  be  a  special  source  of  concern.  As 
noted  above,  there  is  little  empirical  evidence  in  the  literature  concerning 
how  error  enters  into  a  subject's  behavior.  Without  such  evidence,  there 
was  no  choice  except  to  make  some  "seat  of  the  pants"  assumptions.  Specif ;cj! ly , 
it  was  assumed  that  error  enters  in  after  the  stimuli  have  been  combined  and 
that  the  coefficient  of  variation  provides  a  useful  rule-of-thumb  as  to 
the  size  of  the  error  component. ^ 

While  these  assumptions  led  to  reasonable-looking  data,  they  are 
certainly  subject  to  further  analysis.  Additional  simulation  and  empirical, 
research  can  be  used  to  investigate  error  in  more  detail.  Simulations  car.  be 
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employed ,  for  in-nance,  to  explore  alternative  assumptions  alo  .*. 

And  empirical  research,  as  noted  above,  can  be  directed  at  such  issues  as 
the  size  and  location  of  error.  Thus,  the  simulation  approach  taken  here, 
while  not  without  its  problems,  has  raised  some  interesting  questions  and 
suggested  some  new  directions  for  future  research. 


» 
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'''The  data  for  each  simulated  subject  were  created  by  randomly  perturbing 
the  Y'  values  to  generate  two  response  replications,  Y^'  and  Y0 ' .  These  two 
replications  represent  responses  obtained  from  two  independent  presentations 
of  the  stimulus  set  to  a  subject.  The  reason  for  having  two  replications 
is  that  one  of  analyses  required  a  separate  estimate  of  error;  such  estimates 
can  only  be  obtained  by  having  response  replications.  For  analyses  which 
do  not  require  independent  error  estimates,  the  Y^'  and  1  values  were 
averaged  to  produce  Y'  values. 

2 

Ken  Hammond  (personal  communication,  1980)  has  indicated  that  similar 
differences  have  been  observed  in  analyses  run  for  other  purposes.  Moreover, 
the  downward  trend  for  the  adding-model  results  across  negative  intercor¬ 
relations  is  also  commonly  observed.  Thus,  the  present  results  are  apparently 
in  line  with  results  found  in  other  settings. 

3 

To  illustrate  the  effect  that  a  reduced  range  can  have  on  detecting  a 


synergism,  consider  the  following  two  stimulus  sets:  In  set  one,  the  paired 
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cue  values  are  (1,5),  (3,5),  (5,5),  (7,5),  and  (9,5).  In  set  f~,  ire  cue 

values  are  (3,5),  (4,5),  (5,5),  (6,5),  and  (7,5).  Looking  at  just  the 

first  cue,  the  means  are  obviously  equivalent  in  the  two  sets,  but  the 

standard  deviation  for  set  two  is  less.  (Note  that  setting  second  cue 

to  5  is  for  simplicity  and  convenience;  however,  it  does  not  effect  the 

generality  of  the  argument  in  any  way.) 

If  a  multiplying  model  (Equation  2)  is  applied  to  set  one,  the  (error_esw) 

data  will  be  5,  15,  25,  35,  and  45.  If  an  adding  model  (Eq-itior.  3)  is  ..ppliud 

to  the  same  set,  the  data  will  be  6,  8,  10,  12,  and  14.  The  suir.-of-squarec 

2  2  2  2  2 

deviations  between  the  models  is  1  +  7*"  +  15  +23  +31  =  1765,  with  the 

major  difference  between  adding  and  multiplying  occuring  at  the  upper 

end.  For  set  two,  the  data  for  the  multiplying  model  is  15,  20,  25,  30,  and 

35.  The  data  for  the  adding  model  is  S,  9,  10,  11,  and  12.  The  sum-of- 

2  2  2  2  2 

squared  deviations  is  then  7  +11  +15  +19  +23  =  1285,  or  480  less 

than  in  set  one. 

Two  points  are  worth  emphasizing.  First,  the  range  of  the  data  is 
considerably  larger  in  set  one  than  set  two.  This,  of  course,  follows 
directly  the  construction  of  the  stimulus  sets.  Second,  the  difference 
between  adding  and  multiply  is  less  in  set  two  than  in  set  one.  Assuming 
comparable  error  values,  that  implies  that  detecting  the  synergism  in 
set  two  will  be  more  difficult. 

4 

There  is  an  interesting  asymmetrical  relation  between  intercorrelationa* 
values  and  conditional  monotonicity.  If  the  cues  are  positively  correlated, 
then  if  one  cue  is  conditionally  monotone  so  must  be  the  other.  If  the  cues 
are  negatively  correlated,  then  if  one  cue  is  conditionally  monotone  in  one 
direction  the  other  will  be  conditionally  monotone  in  the  opposite  direction. 
Thus,  knowledge  of  the  intercorrelation  value  allows  inferences  to  be  drawn 
about  conditional  monotonicity. 
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However,  know  in;1,  that  end:  cue  is  conditionally  monotone  i...,', 

any  restrictions  on  the  intercorrelation  values.  In  fact,  two  cues  can 
both  be  conditionally  monotone  in  the  same-  direction  and  still  be  highly 
negatively  correlated.  Thus,  statements  about  conditional  monotonicity 
do  not  allow  inferences  to  be  drawn  about  cue  correlations. 

^Factorial  designs  have  frequently  been  criticized  on  the  grounds  that 
(1)  they  are  unrepresentative  (Brunswik,  1956),  and  (2)  they  frequently 
require  too  many  stimulus  cases.  However,  both  of  these  criticisms  can  be 
met  by  the  use  of  fractional  factorial  designs.  Such  partial  designs  allow 
for  both  reduction  in  the  number  of  stimuli  and  control  of  unrepre¬ 

sentative  stimuli.  Some  illustrative  applications  of  fractional  designs 
can  be  seen  in  Phelps  and  Shanteau  (1977)  and  Slovic  (1969). 

It  is  noteworthy  that  most  judgment  researchers  are  aware  of  the 
difficulties  of  nonorthogonal  designs.  For  instance,  the  problem  of 
estimating  weights  with  intercorrelated  cues  is  well  known.  However,  instead 
of  turning  to  orthogonal  designs,  many  researchers  have  used  nonorthogonal 
designs  with  zero  intercorrelations.  As  the  present  results  make  clear, 
the  .00  correlation  condition  shares  many  of  the  same  shortcomings  as  the 
other  conditions.  Also,  it  is  not  clear  how  using  uncorrelated  stimulus 
cue  sets  can  be  any  more  representative  than  using  factorial  designs. 

^As  an  aside,  some  comment  should  be  made  about  the  possibility  of 
normalizing  all  the  data  to  cover  the  same  range.  Normalization  has  the 
advantage  of  allowing  the  same  size  error  term  to  be  used  with  the  three 
data-gencrat ing  models.  Then,  a  single  treatment  of  error  could  be  applied 
to  all  three  data  sets. 

While  this  approach  seems  attractive  in  theory,  there  are  two  practical 
problems.  First,  the  pre-error  data  did  not  cover  any  consistent  range  of 
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values.  That  is,  due  to  randomness  in  the  construction  of  the  stimulus  set 
and  the  restrictions  necessary  to  satisfy  Lhe  constraints  of  intcrcorrelat* 
value,  etc.,  the  various  data  sets  differed  widely  in  their  range.  Thus, 
even  if  the  same  data-gcr crating  model  is  used,  that  is  no  assurance  that 
the  resulting  data  will  1  ave  similar  ranges.  Therefore,  normalization  usi.n 
the  range  (or  any  other  sample  statistic)  would  not  lead  to  equivalent 
data  sets. 

Second,  even  if  the  data  could  somehow  be  normalized  or.  range,  anc.tr.er 
problem  remains.  The  distribution  of  the  data  varies  systematically  be  twee 
the  three  models.  For  the  adding  model,  the  data  is  symmetrically  distribu 
around  the  midpoint  of  the  range.  However,  for  the  multiplying  and  addin*- 
multiplying  models,  the  data  is  skewed  towards  the  lower  end  of  the  range. 
This  asymmetry  is  a  direct  consequence  of  the  multiplying  operation  in  that 
a  high  response  car  only  result  form  the  multiplication  of  two  high  cues; 
otherwise,  relatively  low  responses  will  result.  Thus,  the  use  of  a 
constant  error  term  would  have  dissimilar  effects  on  range-normalized  data 
for  the  three  models.  Proportionally,  the  error  contribution  would  be  less 
for  the  adding  model  than  for  the  other  two  models. 

Because  of  such  difficulties,  it  was  decided  not  to  normalize  the 
data  values.  Instead,  the  error  component  was  individually  calibrated 
to  match  each  data  set.  To  reflect  both  range  variation  and  distributional 
differences,  error  was  made  proportional  to  the  coefficient  of  variation. 
Since  this  coefficient  depends  on  both  the  mean  and  the  standard  deviation, 
it  avoids  most  of  the  problems  outlined  above. 
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Table  2 

Illustrative  Output  From  Stimulus  Construction  Program 


Parameter 

Requested 

Observed 

Specified 

Value 

Value 

Number  of  Cues 

2 

2 

Number  of  Stimulus  Cases 

100 

100 

Minimum  Value,  Cue  1 

1 

6.4 

Maximum  Value,  Cue  1 

100 

100.0 

Minimum  Value,  Cue  2 

1 

2.0 

Maximum  Value,  Cue  2 

100 

97.4 

Mean,  Cue  1 

50 

49.9 

Standard  Deviation,  Cue  1 

20 

19.6 

Mean,  Cue  2 

50 

50.1 

Standard  Deviation,  Cue  2 

20 

19.7 

Chi  Square  Distribution  Test, 

Cue 

1 

— 

12.4 

Significance  Level  (df  =  19), 

Cue 

1 

>.05 

.87 

Chi  Square,  Distribution  Test 

,  Cue 

2 

18.4 

Significance  Level  (df  =  19), 

Cue 

2 

>.05 

.50 

Intercorrelation  Between  Cues 

1  and  2 

0 

o 

o 

Stimulus  Case  Number*3 

Cue  1 

Cue  2 

10 

67.4 

44.3 

20 

45.5 

77.2 

30 

57.6 

24.0 

40 

23.4 

58.7 

50 

76.2 

44.7 

60 

31.2 

12.4 

70 

51.1 

87.0 

80 

36.5 

22.9 

90 

54.8 

19.3 

100 

95.8 

2.0 

aOutput  adapted  from  CUEGEN  program  (see  Kaiser  &  Dickman,  1962;  Naylor, 
et  al,  1965). 

^Every  tenth  case  selected  for  illustrative  purposes. 
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Figure  Co.) Lion;. 

Figure  1.  Average  E-  values  for  the  fit  of  a  linear  regression  mode] 
(Equation  7)  to  three  data-generating  models:  (0)  a  ..mltiplying  model 
(Equation  2),  (1)  an  adding  model  (Equation  3),  and  (2)  an  adding-multiplying 
model  (Equation  4).  (Left  panel  =  25  stimulus  cases,  right  panel  -  100 
stimulus  cases.) 

2 

Figure  2.  Average  R  values  for  the  fit  of  a  multilinear  regression 
model  (Equation  8)  to  three  data-generation  models:  (0)  multiplying, 

(1)  adding,  and  (2)  adding-multiplying.  (Left  panel  =  25  cases,  right 
panel  =  100  cases.) 

o 

Figure  3.  Average  improvement  in  values  for  a  multilinear  regression 
model  over  a  linear  regression  model  for  three  data-generating  models: 

(0)  multiplying,  (1)  adding,  and  (2)  adding-multiplying.  (Left  panel= 

25  cases,  right  panel  =  100  cases.) 

Figure  4.  Average  standardized  regression  weights  (8)  for  crossproduct 
term  in  the  fit  of  a  multilinear  model  to  three  data-generating  models: 

(0)  multiplying,  (1)  adding,  and  (2)  adding-multiplying.  (Left  panel  =  25 
cases,  right  panel  =  100  cases.) 

Figure  5.  Proportion  of  significant  regression  weights  for  crossproduct 
term  in  the  fit  of  a  multilinear  model  to  three  data-generating  models: 

(0)  multiplying,  (1)  adding,  and  (2)  adding-multiplying.  (Left  panel  =  25 
cases,  right  panel  =  100  cases.) 

Figure  6.  Average  F-ratios  for  hierarchical  test  applied  to  three 


data-generating  models:  (0)  multiplying,  (1)  adding,  and  (2)  adding-multiplym 
(Left  panel  =  25  cases,  right  panel  =  100  cases.) 
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l-'igure  7.  1'iuportion  of  significant  hierarchical  tests  fur  three 
data- generating  models:  (0)  multiplying,  (1)  adding,  and  (2)  adding- 
multiplying.  (Left  panel  =  25  cases,  right  panel  =  100  cases.) 

Figure  8.  Average  F-ratios  for  lack-of-fit  test  applied  to  three 
data-generating  models:  (0)  multiplying,  (1)  adding,  and  (2)  adding- 
multiplying.  (Left  panel  =  25  cases,  right  panel  =  100  cases.) 

Figure  9.  Proportion  of  significant  lack-of-fit  tests  for  three 
data-generating  models:  (0)  multiplying,  (1)  adding,  and  (2)  acding-multi?*ying 
(Left  panel  =  25  cases,  right  panel  =  100  cases.) 
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Appendix 

Two  of  the  major  components  of  the  present  simulation  analysis  deserve 
greater  elaboration.  The  first  concerns  specifica.  n  of  the  CUEGEN  program, 
used  to  construct  stimulus  cases.  The  second  involves  discussion  of  the 
various  statistical  procedures  used  to  analyze  the  data.  Each  of  these 
will  be  addressed  in  following  supplementary  materiax. 

CUEGEN  Program 

This  program  is  based  on  procedures  described  in  Kaiser  and  Dickman 
(1962)  with  modifications  outlined  in  Boyle  (1970).  Relevant  programming 
information  can  be  found  in  Naylor,  Balintfy,  Burdick,  and  Chu  (1965). 
Additional  modifications  were  incorporated  especially  for  this  research  proje 
by  Michael  O'Reilly. 

Program  description.  CUEGEN  will  generate  sample  stimulus  cases  which 
will  approximate  user  specified  values  for  the  means,  standard  deviations, 
and  intercorrelations.  The  sample  cases  will  satisfy  with  maximulm  accuracy 
(in  a  least-squares  sense)  the  specified  values  within  the  limits  of  computa¬ 
tional  accuracy,  computing  time,  etc.  The  user  may  also  specify  either  a 
uniform  or  a  normal  distribution  and  this  property  will  also  be  maximally 
satisfied  within  limits. 

The  algorithm  used  by  CUEGEN  starts  with  the  desired  (population)  cor¬ 
relation  matrix.  Through  the  use  of  component  analysis,  random  sample 
matrices  are  generated  from  the  population  correlation  matrix.  If  a  sample- 
matrix  does  not  meet  the  intercorrelation  requirements,  then  adjust¬ 


ments  are  made1  in  the  elements  of  the  matrix  to  bring  it  more  in  line  with 
the  requested  matrix.  Once  a  satisfactory  sample  correlation  matrix  is 
achieved,  linear  transformations  are  applied  to  produce  the  desired  means  and 
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standard  dev  i  at  ion-..  All  properties  ,  i:..- 1  uu  i ■  one  e  ...tr  .  .-i.;  i.u-  , 

are  statistically  clu-.-ked ,  and  a  sample  matrix  is  rejected  if  any  property 
is  not  satisfied  at  tile  .05  3  eve  1 . 

Technical  description.  The  remaining  material  provides  a  more  technical 
discussion  of  the  CLi'.CF.N  program.  A  fundamental  postulate  of  component 
analysis  states  that: 

Z  =  F  X. 

where  F  (or  order  n  x  n)  is  a  factoring  of  R,  the  desired  correlation  matrix, 
and  X  (of  order  n  x  n)  is  a  population  matrix  derived  from  the  components 
in  F.  The  program  begins  by  generating  an  arbitrary  X,  sampling  randomly 
from  uncorrelated  populations  with  any  distribution  and  with  zero  mean  and 
unit  variance.  Then 
Z  =  F  X 

can  be  found,  where  Z  represents  a  matrix  of  observations  from  a  multivariate 
population  with  zero  means,  unit  variances,  and  correlations  R. 

If  the  absolute  correlational  error  is  larger  than  some  acceptable 
value,  then  an  element  of  Z  is  chosen  at  random  and  adjusted  by  a  preset  step 
(default  value  of  1.0)  to  reduce  the  error.  The  direction  of  the  adjustment 
is  chosen  by  examining  the  effects  of  the  change  on  the  correlation  matrix. 
The  process  is  repeated  (within  specifiable  limits)  until  the  desired 
intercorrelation  values  are  obtained.  Finally,  the  rows  of  Z  may,  if 
necessary,  be  linearly  tranformed  to  reflect  specified  means  and  standard 
deviations. 

It  is  possible  for  a  user  to  request:  a  pattern  of  properties  which,  upon 


analysis,  leads  to  a  correlation  matrix  which  is  non-Gramian.  To  produce 
a  correlation  matrix  most  like  the  specified  correlation  matrix,  negative 
eigen  values  (if  any  are  found  during  the  Cv-mian  factoring)  are  set  to  zero. 
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The  program  can  generate  stimuli,:;  cases  having  either  a  u,a .  or;.;  or  a 
>  normal  distribution.  Consequently,  a  test  of  distribution  is  performed  by 

seLting  up  equal  probability  intervals.  A  chi-square  test  is  used  to 
compare  the  actual  number  of  cases  in  each  interval  with  the  expected 
1  number.  Sample  matrices  which  fail  the  distribution  test  are  randomly 

altered  and  reentered  into  the  program  as  necessary  (default  value  of 
5  reentries).  More  information  on  this  or  any  of  the  other  programs  is 
available  upon  request  from  the  author. 

Statistical  Proced urcs 

A  number  of  statistical  procedures  were  evaluated  in  the  researcher 
simulation  stage.  Some  of  these  indices  were  discussed  in  detail  ir.  the 
text  (e.g.,  R^),  while  others  provided  redundant  information  and  so  were 
not  discussed  (e.g.,  r).  In  the  following  material,  the  descriptive  indices 
will  first  be  described  followed  by  the  inferential  indices.  (Except  for 
the  indices  which  are  not  widely  known,  the  computational  formulas  have 
been  omitted.) 

Descriptive  indices.  (1)  Probably  the  simplest  of  the  descriptive  indices 

is  the  ordinary  product  moment  correlation  coefficient ,  r,  between  the  set  of 

predicted  values,  Y,  and  the  set  of  observed  values,  Y'.  (2)  A  closely 

2 

related  measure  is  the  squared  multiple  correlation  value,  R  ,  which  describes 
the  variance-accounted-for  by  a  given  multiple-regression  model.  (3)  Based 
on  the  R^  values  for  the  separate  multiple-regression  models,  a  AR^  can 


be  obtained  from  the  improvement  in  variance-accounted-for  in  going  from 

a  linear  model  (Equation  7)  to  a  multilinear  model  (Equation  8). 

2 

This  reflects  the  increase  in  R  from  adding  a  crossproduct  term  to  a  linear 
model.  (A)  Kor  the  two  regression  models  with  crossproduct  terms  (i.e.  , 
Equations  6  and  8),  the  unstandardized  regres  Ion  weight ,  b,  can  be  obtained 
for  the  x  X2  terms.  (5)  Similarly,  the  standardized  regression  (Beta) 
weight ,  8,  can  be  evaluated  for  the  crossproduct  terms. 
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(6)  Using  the  derived  regression  values,  Y,  the  differcr...,  b*.. •  wven 

derived  and  observed  values,  Y',  can  be  summarized  by  r.ean-ahsol  e.te- 

deviation  (MAD)  scores.  (7)  The  same  comparison  can  be  made  in  terms  of 

root-mean- squared  deviations  (RMSD) .  (8)  The  data  generated  by  the  various 

response  models  can  be  described  by  means ,  and  (9)  standard  deviations. 

(10)  Finally,  a  coef f ieient-of-vari at ion  (standard  deviation  t  mean)  car.  be 

easily  computed  from  the  preceeding  measures. 

Inferential  indices.  (11)  The  sign  if  i can.ce  of  the  s  :nd..raizeG  regression 

weight  (computed  in  step  5)  can  be  determined  for  the  crossproduct  terms  in 

Equations  6  and  8.  (12)  The  hierar chica 1  multiple  regression  approach  (Cohen  L 

2 

Cohen,  1975,  chapter  8)  is  based  on  testing  the  improvement  in  R  in  going 
from  a  linear  model  to  a  multilinear  model  (see  step  3).  The  computation 
formula  used  was: 

F  =  ^r28  "  r27)  -  k.7  -  kg  -  1) 

(1  -  r28)  kg 

where  R2g  and  kg  refer  to  the  variance-accounted-for  and  number  of  independent 

variables  added  (=1),  respectively,  for  the  multilinear  model  in  Equation  8. 

2 

R  7  and  k^  (=2)  are  the  comparable  values  in  Equation  7.  The  F-ratio,  based 
on  kg,  (N  ~  ^7  -  kg  -  1)  degrees  of  freedom,  can  be  tested  directly  for 
significance  (see  Arnold  &  Evans,  1979,  p.  AA). 

(13)  The  lack-of-f it  test  (Draper  &  Smith,  1966,  sec.  1.5)  is  based 
on  splitting  the  residual  sum-of-squares  (SS)  in  a  regression  analysis  into 
two  parts:  a  lack-of-fit  SS  and  a  "pure  error"  SS.  The  F-ratio  for  lack-of- 
fit  can  be  computed  directly  following  standard  analysis-of-variance  logic. 

(1A)  To  obtain  an  estimate  of  pure  error ,  a  complete  repliation  of  the  data 
was  generated.  The  difference  between  the  replications,  '  and  Y2' ,  provides 
an  independent  error  estimate  for  use  in  tne  lack-of-fit  test  (step  13).  For 
all  other  analyses,  the  difference  in  the  replicates  was  ignored  (see  footnote 


1  for  further  details). 
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